Skip to content

fix(gp): bless live response oracles for 9 unblessed scripts + gate future regrowth - #229

Merged
mikemcdougall merged 6 commits into
trunkfrom
fix/202-gp-response-oracles
Sep 14, 2026
Merged

mikemcdougall merged 6 commits into
trunkfrom
fix/202-gp-response-oracles

Conversation

@mikemcdougall

@mikemcdougall mikemcdougall commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Stacked on #228 (its current head 7309939), because trunk's ephemeral-server-smoke is red. The unpinned seed inserts services.max_record_count, but the nightly-aot image schema has no such column, so the seed fails with psql exit 3. Once #228 merges, this diff shrinks to its two commits.

Problem (#202)

ephemeral-server-smoke graded no response oracle for 9 of the 25 supported honua-gp eval scripts:

  • describe_segments[_fields]
  • list_fields_segments[_filtered|_wildcard]
  • insert_cursor_append_rows
  • update_cursor_close_status
  • update_cursor_delete_closed
  • make_table_view

A supported script with no oracle counted as a silent "unblessed" pass. So a regression in Describe/ListFields or in the write cursors could pass this lane unnoticed. --require-supported-pass-rate 1.0 did not help, because unblessed scripts counted as passing.

Changes

Commit 1 (the earlier revision, rebased onto #228). Blessing the cursor scripts exposed a real bug. honua_gp.da read only .attributes, but the live SDK's QueryFeature exposes .properties/.id. As a result, every live SearchCursor/UpdateCursor field read, OID included, returned None. This commit:

  • fixes that bug, with regression tests against QueryFeature;
  • adds schema oracles for Describe/ListFields;
  • adds apply-edits oracles for the cursor scripts.

Commit 2 (this revision). Commit 1 still left three gaps:

  • update_cursor_delete_closed was blessed to delete_count: 0. An earlier script had already archived every CLOSED row, so it deleted nothing, and a broken deleteRow would have passed.
  • The write-cursor oracles recorded only applyEdits counts, never the dataset contents afterwards.
  • make_table_view still passed through an allow-list as "unblessed".

Commit 2 closes them:

  • Insert. insert_cursor_append_rows reads back the object ids the server assigned and records the rows that persisted.

  • Update and delete. update_cursor_close_status and update_cursor_delete_closed each insert their own fixture row and scope the edit to it by name. They then read back:

    • the value the update left;
    • that the deleted id is the inserted id;
    • that zero rows remain.

    Neither depends on script order or on rows left by an earlier run.

  • Table view. make_table_view applies status = 'active' and records the rows the server returns through the view (GetCount + SearchCursor).

  • Gate. The allow-list is removed. In live mode, a supported script without a response oracle now always fails. A new unit test, test_every_supported_script_has_a_response_oracle, fails the stub lane as soon as a supported golden lacks one.

  • Generator. _generate_scripts.py now owns these four scripts' oracles. Before this change, regenerating the scripts silently dropped the hand-edited emit lines.

Per-script decision

All 9 scripts have an observable result, so the unblessed set is now empty.

Script Oracle Where the expected value comes from
describe_segments, describe_segments_fields, list_fields_segments 15 fields with names and types, OID field objectid, shape Point, SRID 4326 honua.layer_fields for layer 0 in tests/seed/client-compat-v1.sql at 7ba4226
list_fields_segments_filtered / _wildcard the 6 String fields / status the same seed rows, filtered
insert_cursor_append_rows add_count 3 and the persisted rows [CLOSED, Side Rd], [OPEN, Elm Ave], [OPEN, Main St] the rows the script writes
update_cursor_close_status update_count 1, the updated row reads [ARCHIVED, Close Status Rd], closed_remaining 0 the script's own fixture row
update_cursor_delete_closed delete_count 1, deleted_inserted_row true, rows_remaining 0 the script's own fixture row
make_table_view view_count 5, rows alpha, epsilon, eta, gamma, iota (all active) the seed's status = 'active' rows

Evidence

All live runs used the pinned candidate: ghcr.io/honua-io/honua-server@sha256:dd50cd81… with the seed at 7ba4226. The stack was started with the workflow's own compose override (migrate first, then seed, then the SpatialJoin join layer).

  • Fresh stack, committed goldens, at head 3b46f68: honua-gp eval [live]: 52/52 passed (100%); supported 27/27 (100%).

  • Bless run on a fresh stack: only the four scripts changed in this revision produced new goldens. The other 48 matched their committed goldens.

  • Rerun against the already-mutated database: all four changed scripts pass again. The older seed-pinned count scripts drift there as expected, as docs/golden-eval.md already documents.

  • Negative control: I reverted only the .properties fallback in _attrs_for_feature and ran just the four changed scripts live. The result was supported 0/4:

    • update_cursor_delete_closed got delete_count 0, rows_remaining 1. The old count-only oracle blessed exactly that as a pass.
    • make_table_view got [[null, null] × 5].
    • The two remaining cursor scripts also failed their read-back values.

    I restored the file afterwards and the worktree was clean.

  • Unit tests: python -m pytest packages/honua-gp/tests -q gives 281 passed, 9 skipped.

  • Other checks: stub eval 52/52; python -m honua_gp._cli matrix --check packages/honua-gp/docs/compatibility-matrix.md passes.

  • Out of scope: ArcGIS Pro output parity is not claimed. The oracles are server-observed values on the client-compat seed, as described in docs/golden-eval.md.

Cross-links

Closes #202

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

mikemcdougall and others added 2 commits September 13, 2026 23:16
…moving canary

The blocking live GP smoke lane consumed the moving nightly-aot tag and an
unpinned honua-server seed checkout, so a commit's verdict depended on what
honua-server merged that day.

- pinned (push/pull_request): image digest + seed revision of the 2026.1-rc.2
  governed candidate (honua-server@7ba4226, sha256:dd50cd81...), checked
  against the pulled image's digest and revision label; seed checked out at
  that exact revision. Documented promotion path in the job comment.
- canary (daily schedule / dispatch mode=canary): resolves nightly-aot to a
  digest + revision and seeds from it; every step is non-fatal so the
  check-run never concludes failure; the verdict creates/updates the
  "honua-gp moving canary drift" issue.
- a verdict step separates environment-startup-failure, gp-correctness-failure,
  eval-incomplete and not-executed from passed, records smoke-identity.json,
  and a missing compose fixture now fails instead of skipping green.
- the persistent HONUA_GP_EVAL_BASE_URL target is honoured on manual dispatch
  only; the canary has its own concurrency group.

Closes #201
…uture regrowth

Live smoke graded no response oracle for 9 of 25 supported honua-gp eval
scripts (Describe x2, ListFields x3, InsertCursor, UpdateCursor x2), so a
response-parsing regression in schema introspection or write-cursor round
trips would pass the lane silently as an "unblessed" pass.

Blessing the two write-cursor scripts against a fresh seed surfaced a real
bug rather than a missing oracle: honua_gp.da._values_for_row/_extract_oid
only recognized a feature's .attributes, but the live SDK's
honua_sdk.models.QueryFeature exposes GeoJSON-shaped .properties/.id instead
-- SearchCursor/UpdateCursor silently returned None for every field (OID
included) against a real server. Stub-mode CI never caught this because the
eval stub's _StubFeature happens to have .attributes. Fixed both helpers to
read the real feature shape (preferring the SDK's already-resolved .id for
OID@, since the seed's object-id field is lower-case objectid and never
matched the existing OBJECTID/oid/FID key list), added regression tests
against the actual QueryFeature type, and re-blessed against a fresh stack
-- update_cursor_close_status now genuinely proves an update round-trip
(update_count: 1) instead of masking the bug behind a vacuous zero.

Describe/ListFields (schema fetch) and InsertCursor were already correct;
those got a schema/apply-edits fingerprint oracle. MakeTableView makes no
server round trip at all (a client-side session alias registration) -- no
response to fingerprint, so it goes in a new UNBLESSED_ALLOWLIST.json with
the reason instead of a silent pass.

run_eval.py now fails a live-mode supported script that has no response
oracle and no UNBLESSED_ALLOWLIST.json entry, so this set cannot regrow
unnoticed (verified: an emptied allow-list correctly fails make_table_view.py
with the new reason).

Closes #202
…view

The write-cursor oracles recorded applyEdits counts only, and
update_cursor_delete_closed was blessed to a vacuous delete_count 0: an
earlier script had already archived every CLOSED row, so a broken
deleteRow passed. make_table_view still passed as an allow-listed
"unblessed" script.

- insert_cursor_append_rows reads the inserted object ids back and records
  the persisted rows.
- update_cursor_close_status / update_cursor_delete_closed insert their own
  fixture row, scope the edit to it by name, and record the value left
  behind / that the deleted row is gone. They no longer depend on script
  order or on a previous run's rows.
- make_table_view applies a where clause and records the rows the server
  returns through the view (the five 'active' seed rows).
- The allow-list is gone: a live supported script without a response oracle
  fails, and a unit test fails the stub lane when a supported golden has
  none.
- The generator now owns these scripts' oracles, so regenerating no longer
  drops them.
@mikemcdougall

Copy link
Copy Markdown
Contributor Author

Rebuilt head 3b46f68. It sits on #228's current head 7309939 and supersedes 96ec5c8, which had the same oracle commit rebased onto trunk without #228's pin, so ephemeral-server-smoke would stay red. It adds commit 3b46f68: write cursors and make_table_view now read back dataset contents, the non-verifying delete_count: 0 oracle is gone, and so is the allow-list. Live evidence is in the PR description.

@mikemcdougall
mikemcdougall merged commit 5ad6c66 into trunk Sep 14, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live smoke grades no response oracle for 9 of 25 supported eval scripts (unblessed passes silently)

1 participant